1. Field of the Invention
The invention relates to an apparatus allowing the composition of two images according to the so-called xe2x80x9cblue-screenxe2x80x9d method.
2. Description of the Prior Art
As disclosed in U.S. Pat. No. 4,100,569, one of the two imagesxe2x80x94henceforth also called the foreground imagexe2x80x94here shows an object or a three-dimensional scene with a substantially monochrome, preferably blue, background. The difference in color allows to separate the object or three-dimensional scene and the blue background.
The other imagexe2x80x94henceforth also called the background imagexe2x80x94on the other hand shows an arbitrary background into which the object or three-dimensional scene of the foreground image is to be transferred. This way it is possible to, for example, film a TV moderator with a blue background in a TV Studio, and to subsequently transfer this scene into an arbitrary setting, recorded previously as a separate image.
To this end, the known apparatus calculates two control parameters for each scanning point based on the chromatic values of the foreground image, wherein the first control parameter (xe2x80x9cbackground control voltagexe2x80x9d) specifies how intensely the background image is to be shown in the resulting imagexe2x80x94henceforth also called the composite image, whereas the second control parameter (xe2x80x9cclamping voltagexe2x80x9d) specifies to which degree the color of the blue background must be attenuated in the composite image in order to suppress the unwelcome color contribution of the blue background.
The composite image is then combined by a TV mixer from the chromatic values of the background image weighted by the first control parameter on the one hand, and from the chromatic values of the foreground image on the other hand, wherein the blue color component of the foreground image is attenuated according to the second control parameter.
It is important in this context that the color components of the foreground image are taken into account in each scanning point in the mixing of the composite image. Thus, the foreground image is not subject to a switching process assembling the composite image of either the chromatic values of the foreground image or of the chromatic values of the background image. On the other hand, this allows to also reproduce semi-transparent objects in the foreground image, which in the composite image have the new background shine through instead of the blue background. On the other hand, this way the optical resolution of the foreground image is totally conserved.
The more recent U.S. Pat. Nos. 5,424,781; 5,343,252; 4,625,231; 4,589,013 and 4,344,085 describe sophistications of the method described above, wherein essentially the formulas for determining the two control parameters are modified by taking into account several correction factors. The object is to produce an optically satisfying composite image even for difficult color compositions of the foreground image.
The methods disclosed in the above documents thus allow the composition of two images to give composite image according to the xe2x80x9cblue screenxe2x80x9d method, however, especially for certain spectral compositions of the foreground image, they result in a composite image that appears optically unnatural. For professional use, e.g. to produce special effects in the film industry, this makes a costly manual touching up of the composite image necessary.
Additionally, the documents cited above, particularly U.S. Pat. No. 4,589,013, disclose the so-called xe2x80x9cchroma-keyxe2x80x9d technique. This technique also renders the composition of two images to give a composite image. However, as opposed to the method described in the beginning, with the chroma-key technique the individual scanning points in the composite image are composed either of the color components of the foreground image or of the color components of the background image. Thus, the foreground image is subject to a switching process. Because of the analog image encoding this results in an accordingly reduced optical resolution and complicates a reproduction of semi-transparent objects.
Accordingly, the object of the invention is to create an apparatus and/or a method to combine a foreground image and a background image to give a composite image with a natural optical appearance, wherein the foreground image pictures an object or a scene with a substantially monchrome background.
The invention is based upon the insight that the deterioration of the optical resolution, usually entailed by composing the composite image either of the pictorial content of the foreground image or of the pictorial content of the background image, no longer ensues, if the image processing is done digitally and if the separation of the foreground image into object on the one hand and monochrome background on the other hand is done for each pixel individually.
The present invention thus covers the art of providing both the foreground image and the background image in digital form, and of composing the composite image in each pixel either of the chromatic values of the corresponding pixel of the foreground image or of the chromatic values of the corresponding pixel of the background image, the composite image being subsequently processed by an image processing unit in order to reproduce e.g. transparency effects.
The term chromatic value as used here and below is to be interpreted broadly and just means that the pictorial appearance of a pixel can be calculated from its chromatic values(s). In the preferred embodiment of the invention, however, the chromatic values represent the intensity of the three primary colors red, green and blue, following the well known RGB color scheme. They are represented by numbers between 0 and 255, a high number standing for a high intensity of the corresponding color component.
Also the term monochrome background with which the object or scene in the foreground image is depicted is to be interpreted broadly. Monochrome does not imply each pixel in the region of the background has exclusively one color component, whereas the other two color components are zero. Decisive is rather the predominance of the intensity of one componentxe2x80x94preferably bluexe2x80x94over the other two color components as well as that the color composition within the image region forming the background varies as little as possible.
A first processing unit subdivides the foreground image into object or scene and monochrome background. It calculates a control signal for each pixel, based both on the chromatic values of the foreground image and on predetermined chromatic values representing the color composition of the monochrome background. The control signal reflects whether a given pixel is part of the object and/or scene or part of the monochrome background.
A selection unit assembles the composite image. It calculates the chromatic values of each pixel, based on either the chromatic values of the corresponding pixel of the foreground image or on the chromatic values of the corresponding pixel of the background image.
When subdividing the object and/or scene and the background in the foreground image it is important that the predetermined chromatic values, representing the color composition of the monochrome background, reflect the actual color composition of the monochrome background as accurately as possible. However, this is difficult to accomplish since the foreground image often is available as a digitized photography only. Thus, the user has no information on the color composition of the monochrome background utilized in the photography.
In one embodiment of the invention it is therefore intended to establish the color composition of the monochrome background by calculating the average of the individual color components within the entire foreground image. The numbers thus calculated are in rather good agreement with the actual chromatic values of the monochrome background especially if the object or scene takes only little space in the foreground image, while the greater part of the foreground image is taken up by the monochrome background. However, because these average values always differ somewhat from the actual chromatic values of the monochrome background, according to one embodiment an input unit is provided via which the user, starting from the above average values, may change the predetermined chromatic values in order to achieve a better separation of object and/or scene and monochrome background.
In the preferred embodiment of the invention, the control signal, calculated by the first processing unit and allowing to distinguish in each pixel object and/or scene and monochrome background, is stored in a storage matrix. This allows to later process the composite image separately for object and/or scene and new background. Each element of the storage matrix is uniquely assigned to one pixel. For example, it would be set to xe2x80x9c1xe2x80x9d if the pixel is part of the monochrome background, and it would be set to xe2x80x9c0xe2x80x9d, if the pixel depicts a part of the object or the scene.
According to the invention, after calculating the chromatic values of the individual pixels of the composite image, the composite image is processed further in order to achieve as natural a pictorial appearance of the composite image as possible. To this end, the apparatus comprises a second processing unit. This unit calculates correction values for each pixel of the composite image, based on the chromatic values of the corresponding pixel of the foreground image and/or on the chromatic values of the corresponding pixel of the background image. The actual processing is subsequently done by an image processing unit linked to the second processing unit. This image processing unit changes the chromatic values of the individual pixels of the composite image according to the correction values determined by the second processing unit, in order to obtain a natural pictorial appearance.
In an additional version of the invention, worth to be protected in its own right, it is intended to take into account the chromatic effect of the background image on the object or the scene. For example, if the object is transferred with unvaried color into a strongly reddish background image, it will appear unnatural in the new background because of the great contrast in color. Hence, in order to obtain a natural pictorial appearance it is of advantage to adapt the color composition of the object and/or the scene to the color composition of the background image. For the example described above where an object is transferred into a reddish background, this entails increasing the intensity of the red color component in the region of the image occupied by the object.
In this version of the invention the second processing unit determines correction values for each pixel of the composite image in that area of the image occupied by the object and/or the scene. The correction values reflect the chromatic effect of the background image on the object and/or the scene.
In a preferred embodiment of this version it is intended to store the correction values in a storage matrix. Each element of the storage matrix is uniquely assigned to one pixel of the composite image. By a separate pictorial representation of the contents of this storage matrix the user can get an overview of the chromatic effect of the background image on the transferred object, without examining the composite image in all detail.
Here, the correction values are preferably calculated considering the spatial distribution of color in the background image into which the object or the scene of the foreground image is transferred. For example if the object is placed in the middle of the background image, and the composite image is reddish on the right side, and more greenish on the left side, then in order to achieve a natural chromatic appearance of the object it makes sense to shift the color composition of the object towards red on the right side, and towards green on the left side.
To do so, the second processing unit subdivides the area of the composite image filled by the background image in a plurality of smaller image areas. For each of these areas it calculates representative chromatic values by, e.g., averaging the individual color components of all pixels in the respective image area. When calculating the correction values for an individual pixel, the chromatic effect of each image area is considered separately, the chromatic effect being weighted by the distance between the pixel and the image area considered. This means when calculating the correction value of a given pixel, a very distant pixel has little influence. On the other hand, pixel in the direct neighborhood has a relatively great influence on the chromatic effect of the pixel under consideration. Above procedure leads to a natural chromatic appearance of the object and/or the scene in the background image.
In an additional version of the invention, worth to be protected in its own right, it is intended to account for transparency effects in the calculation of the correction values for the composite image. Consider, e.g., a bottle of greenish glass to be transferred into a reddish background image. In the composite image, this bottle must be depicted with a slightly yellowish hue because of the reddish background shining through the bottle.
To take into account transparency effects it is therefore necessary to admix the background image to the composite image in the transparent image region.
It also is necessary to suppress the unwelcome color distortion caused by the monochrome background shining through the transparent object of the foreground image.
In this version of the invention the second processing unit therefore calculates for each pixel of the foreground image a transparency value based on its chromatic values reflecting the transparency of a given pixel. Here, one uses the fact that the color components of a transparent object are changed according to the color composition of the monochrome background.
The transparency value is subsequently fed to a TV-mixer. The mixer on the one hand adds the pictorial information of the background image to the composite image in the image region occupied by the transparent object or scene of the foreground image. On the other hand it suppresses the color distortion caused by the monochrome background. To do so, the color components of the composite image are diminished in the image region depicting a transparent object according to the color composition of the monochrome background.
In a preferred embodiment of this version the transparency value is stored in an extra storage matrix. By pictorially representing the contents of the storage matrix a fast survey of the influence of transparency effects is possible.
According to an additional version of the inventions, worth to be protected in its own right, it is intended to account for the shadow formation in the composite image caused by the object or the scene on the background image. Normally the object recorded with a monochrome background casts a shadow on the backing. This shadow is not reproduced accordingly in the composite image because of the separation of object on the one hand and monochrome background on the other hand.
In this version of the invention it is thus intended that a correction value is determined by the second processing unit for each pixel of the composite image in the area occupied by the background image, based on the chromatic values of the corresponding pixel in the foreground image. This correction value represents the effect of shadow formation in the composite image.
The correction value is subsequently fed to the image processing unit. To depict shadow formation, this unit darkens the composite image as a function of the correction value. In a preferred embodiment of this version an input unit is provided. This unit is linked to the image processing unit.
It allows the input of a control value which controls the harshness of the shadow formation in the composite image. This way the user can tune continuously the harshness of the shadows cast by the object or the scene on the background image in the composite image.
In an advantageous embodiment of this version it is intended to store the correction values reflecting the intensity of the shadow formation in a storage matrix. Each element of the storage matrix is uniquely assigned to one pixel of the composite image. By pictorially representing the contents of this storage matrix the user can survey the shadow distribution without depicting the composite image in all detail.
However, the object, when recorded with a monochrome background not only casts shadows on the background. It also produces illuminations due to reflections at the object surface. This effect must in particular be taken into account for shiny objects.
In a developed version of the invention, worth to be protected in its own right, it is thus intended to also account for reflections at the object or the scene in the composite image by brightening the composite image in appropriate areas.
To achieve this goal, the second processing unit determines for each pixel of the composite image in the image area occupied by the background image a correction value, based on the chromatic values of the corresponding pixels of the foreground image. This correction value reflects the illuminations of the monochrome background due to reflections at the object.
This correction value is subsequently fed to the image processing unit which brightens the composite image accordingly, depending on the correction value, in order to represent the illuminations. In the preferred embodiment of this version an input unit is provided, linked to the image processing unit, and allowing the input of a control value which controls the degree of illumination of the composite image. This way the user can control in the composite image the degree to which reflections at the object of the foreground image are to be reproduced in the composite image.
In one embodiment of this version it is intended to store the correction value describing the illuminations of the monochrome background of the foreground image due to reflections at the object in a storage matrix. Each element of the storage matrix is uniquely assigned to one pixel of the composite image. By pictorially representing the contents of this storage matrix the user can in a simple way get an overview of the illuminations due to reflections at the object of the foreground image, without depicting the composite image in all detail.
The above apparatus is particularly well suited for implementation in the context of a parallel computer architecture, because the chromatic values of the individual pixels of the composite image can be determined independently and thus in parallel.
However, the invention is not restricted to an implementation in the context of microprocessor systems in which the above described operations proceed controlled by a program. It may also be implemented by hardware only.